Rozpoznawanie zapisu nutowego 
na podstawie materiałów i kodu
Serious and Not-So-Serious Image Processing Applications Markus van Almsick 0Markus van Almsickpodstawie

Wyprostowanie pięciolinii09

Będziemy pracowali na poniższym skanie:

musicScan = ;
ImageDimensions[musicScan]

Opening[musicScan,BoxMatrix[{1,0}]]

Poniżej została zastosowana transformacja morfologiczna BottomHatTransform[] jest to operacja zamknięcia na obrazie z elementem strukturalnym BoxMatrix[] odjąć cały obraz

BottomHatTransform[musicScan,BoxMatrix[{1,0}]]

Poniżej zastała użyta funkcja ImageLines[], która znajduje linie proste z grafice i zwraca współrzędne początka i końca linii

lines =ImageLines[BottomHatTransform[musicScan,BoxMatrix[{1,0}]],Automatic,0.0065]
Dimensions[lines]

φ=ArcTan[(lines[[1,2,2]]-lines[[1,1,2]])/596]

Prostujemy pięciolinię:

alignedMusicScan=ImageRotate[musicScan,-φ , Background-> White]

staffLines =ImageLines[BottomHatTransform[alignedMusicScan,BoxMatrix[{1,0}]],Automatic,0.0065]

Show[alignedMusicScan,Graphics[Prepend[Map[Line,staffLines],Red]]]

Wydrębnienie pięciolinii

Poniżej wyznaczenie odległości pomiędzy liniami pięciolinii:
Wyznaczamy współrzędne linii

staffLines[[All,All,2]]

Sortujemy

Sort@Map[Mean,staffLines[[All,All,2]]]

i wyznaczamy różnicę pomiędzy dwiema sąsiadującymi liniami

Differences@Sort[Map[Mean,staffLines[[All,All,2]]]]
ListPlot[%]

Δ =Median@Differences@Sort@Map[Mean,staffLines[[All,All,2]]]

staffLineGroups=Gather[Sort[staffLines,(Last@Mean@#1>Last@Mean@#2)&],(Last@Mean@#1-Last@Mean@#2<5Δ)&]

originalStaffs=
Map[
ImageTrim[alignedMusicScan,Flatten[#,1],{0,4Δ}]&,
staffLineGroups
];
Dimensions[%]

TableForm[originalStaffs]
originalStaffs[[1]]

staff=originalStaffs[[1]]

Dilation[staff,BoxMatrix[{Δ/6,0}]]

ColorNegate@BottomHatTransform[staff,BoxMatrix[{0,Δ}]]

imageMax[imgs__Image]:=ImageApply[Max,ColorCombine[{imgs}]];
imageMin[imgs__Image]:=ImageApply[Min,ColorCombine[{imgs}]];

withoutStaffLines=
imageMin[
Closing[staff,BoxMatrix[{Δ/6,0}]],ColorNegate@BottomHatTransform[staff,BoxMatrix[{0,Δ}]]
]

imageMax[
withoutStaffLines,
MorphologicalBinarize[withoutStaffLines,{0.66,0.33}]
]

removeStaffLines[img_,Δ_]:=
With[
{withoutStaffLines=
imageMin[
Closing[img,BoxMatrix[{Δ/6,0}]],
ColorNegate@BottomHatTransform[img,BoxMatrix[{0,Δ}]]
]},
imageMax[
withoutStaffLines,
MorphologicalBinarize[withoutStaffLines,{0.66,0.33}]
]
]

Symbole muzyczne

Row@Riffle[{,,,,,}," "]

Row@Riffle[{,,,,},"  "]

δ = 8;
resizeFactor = 1/Round[Δ/δ,0.125];
staffs = 
Map[
ImageResize[#,Scaled[resizeFactor]]&,
originalStaffs
];
symbolRows=
Map[
ImageResize[removeStaffLines[#,Δ],Scaled[resizeFactor]]&,
originalStaffs
];

TableForm[symbolRows]

Wyszukiwanie wzorca symboli muzycznych

Wyszukujemy wzorca symbolu myzycznego przy pomocy korelacji dwóch grafik ImageCorrelate[]

detectMusicSymbol[img_Image,symb_Image,threshold_?NumberQ]:= 
ColorNegate@Binarize[
ImageCorrelate[img,symb,EuclideanDistance],
threshold
]

staffs[[1]]
detectMusicSymbol[staffs[[1]],,6.5]

findMusicSymbolPositions[img_Image,symb_Image,threshold_?NumberQ]:= 
ComponentMeasurements[
detectMusicSymbol[img,symb,threshold],
"Centroid"
][[All,2]]

findMusicSymbolPositions[staffs[[1]],,6.5]

findMusicRests[img_Image,symb_Image,threshold_?NumberQ]:=
With[
{center = Last@ImageDimensions[img]/2},
Select[
findMusicSymbolPositions[img,symb,threshold],
(center-3<Last[#]<center+3)&
][[All,1]]
]

Wyodrębnienie pauz

findFullRests[img_Image]:=Map[(#->SoundNote[None,1])&,findMusicRests[img,,6.0]]
findHalfRests[img_Image]:=Map[(#->SoundNote[None,1/2])&,findMusicRests[img,,6.0]]
findQuarterRests[img_Image]:=Map[(#->SoundNote[None,1/4])&,findMusicRests[img,,6.6]]
findEighthRests[img_Image]:=Map[(#->SoundNote[None,1/8])&,findMusicRests[img,,6.5]]

findRests[img_Image]:=Sort@Join[findFullRests[img],findHalfRests[img],findQuarterRests[img],findEighthRests[img]]

staffs[[1]]

findRests[staffs[[1]]]

staffs[[4]]

findRests[staffs[[4]]]

Wyodrębnianie znaków chromatycznych (bemol, kasownik, krzyżyk)

findFlats[img_Image]:=findMusicSymbolPositions[img,,5.1]
findNaturals[img_Image]:=findMusicSymbolPositions[img,,4.6]
findSharps[img_Image]:=findMusicSymbolPositions[img,,5.1]

symbolRows[[1]]

findFlats[symbolRows[[1]]]

Wyodrębnianie znaków nut

naturalPitches = Flatten@Table[Map[#<>ToString[k]&,{"C","D","E","F","G","A","B"}],{k,1,6}]
Dimensions[naturalPitches]

toNaturalPitch[y_]:=naturalPitches[[Round[2/δ y-0.375`]+16]]

Poniżej obniżenie i podwyższenie zapisu o pół tonu

makeSharp=StringReplace[#,pitch:CharacterRange["A","G"]:> pitch<>"♯"]&;
makeNatural = StringReplace[#,pitch:CharacterRange["A","G"]~~{"♯","♭"}:> pitch]&;
makeFlat=StringReplace[#,pitch:CharacterRange["A","G"]:> pitch<>"♭"]&;

Poniżej wyznaczenie pozycji krzyżyka

symbolRows[[4]]
findSharps[symbolRows[[4]]]

Poniżej znajdujemy wysokość na której znaduje się krzyżyk (wynik można uzyskać w Mathematice na przynajmniej dwa sposoby)

Last@First@findSharps[symbolRows[[4]]]

findSharps[symbolRows[[4]]][[1,2]]

toNaturalPitch[37]

makeSharp@toNaturalPitch[Last@First@findSharps[symbolRows[[4]]]]

Klucz wiolinowy

findGClef[img_Image]:= 
With[
{center = Last@ImageDimensions[img]/2},
Select[
findMusicSymbolPositions[img,,14.0],
(center-1<Last[#]<center+5)&
][[All,1]]
]

clef= findGClef[staffs[[1]]]

flats = findFlats[symbolRows[[1]]]

minorPreamble=First@Split[Sort@Prepend[flats,clef+3/2δ],(First[#2]-First[#1]<3/2δ)&]

minor = {"B"-> "B♭","E"-> "E♭", "A"-> "A♭","D"-> "D♭","G"-> "G♭","C"-> "C♭", "F"-> "F♭"};

Take[minor,Length@minorPreamble-1]

Ceiling[Max[minorPreamble[[All,1]]]+δ/2]

findStaffSignature::noGClef = "No G-clef detected.";

findStaffSignature[staff_Image,symb_Image]:=
Module[
{clef= findGClef[staff],
sharps = findSharps[symb],
flats = findFlats[symb],
majorPreamble,minorPreamble,
major = {"F"-> "F♯","C"->"C♯","G"->"G♯","D"->"D♯","A"-> "A♯","E"-> "E♯","B"-> "B♯" },
minor = {"B"-> "B♭","E"-> "E♭", "A"-> "A♭","D"-> "D♭","G"-> "G♭","C"-> "C♭", "F"-> "F♭"}},
If[clef ==={},Message[findStaffSignature::noGClef];Return[$Failed]];
majorPreamble=First@Split[Sort@Prepend[sharps,clef+3/2δ],(First[#2]-First[#1]<3/2δ)&];
minorPreamble=First@Split[Sort@Prepend[flats,clef+3/2δ],(First[#2]-First[#1]<3/2δ)&];
{Ceiling[Max[{majorPreamble[[All,1]],minorPreamble[[All,1]]}]+δ/2],
Join[Take[major,Length@Rest@majorPreamble],Take[minor,Length@Rest@minorPreamble]]}
]

{offset,signature}=findStaffSignature[staffs[[1]],symbolRows[[1]]]

Wartości nut

staffs[[4]]

shortStaff = ImageTake[staffs[[4]], All, {offset, -1}]

shortSymbolRow=ImageTake[symbolRows[[4]],All,{offset,-1}]

BoxMatrix[{δ,0}]
BoxMatrix[{0,δ/6}]

Thinning@Binarize@BottomHatTransform[Closing[shortStaff,BoxMatrix[{δ,0}]],BoxMatrix[{0,δ/6}]]

noteStemBBs=ComponentMeasurements[Thinning[Binarize[BottomHatTransform[Closing[shortStaff,BoxMatrix[{δ,0}]],BoxMatrix[{0,(δ/6)}]]]],"BoundingBox"][[All,2]]

noteStemBBs=Select[noteStemBBs,(2.33 δ<#[[2,2]]-#[[1,2]] )&]

sharps=findSharps[shortSymbolRow]

naturals=findNaturals[shortSymbolRow]

flats=findFlats[shortSymbolRow]

accidentalPositions = Join[sharps,naturals,flats][[All,1]]

noteStemBBs=Pick[noteStemBBs,Map[(Min[Abs[accidentalPositions-#]]>2δ/3)&,Map[Mean,noteStemBBs[[All,All,1]]]]]

findNoteStemBoundingBoxes[img_Image, symb_Image]:=
Module[
{accidentalPositions,noteStemBBs},
accidentalPositions=Join[findSharps[symb],findNaturals[symb],findFlats[symb]][[All,1]];
noteStemBBs=ComponentMeasurements[Thinning@Binarize@BottomHatTransform[Closing[img,BoxMatrix[{δ,0}]],BoxMatrix[{0,δ/6}]],"BoundingBox"][[All,2]];
noteStemBBs=Select[noteStemBBs,(2.33 δ<#[[2,2]]-#[[1,2]] )&];
Pick[noteStemBBs,Map[(Min[Abs[accidentalPositions-#]]>2δ/3)&,Map[Mean,noteStemBBs[[All,All,1]]]]]
]

imageMask[{elems__},img_Image]:=
 Rasterize[
Graphics[{White,elems},Background-> Black, PlotRange-> Transpose[{{0,0},ImageDimensions[img]}]],
"Image",
ColorSpace->"Grayscale" ,
ImageSize-> ImageDimensions[img]
]

HighlightImage[shortStaff,imageMask[Apply[Rectangle[#1-2,#2+2]&,noteStemBBs,{1}],shortStaff]]

findFullNoteHeads[symb_Image]:=detectMusicSymbol[symb,,4.5]

findHalfNoteHeads[symb_Image]:=detectMusicSymbol[symb,,4.5]

makeNotes[signature_List][{t_,y_},flags_Integer]:=t->SoundNote[StringReplace[toNaturalPitch[y],signature],(1/(2^(flags+2)))]

findFullNotes[symb_Image, signature_List]:=
Map[
makeNotes[signature][#,-2]&,
Sort@ComponentMeasurements[findFullNoteHeads[symb],"Centroid"][[All,2]]
]

lowerNoteMask[img_Image,stemBBs_List] := imageMask[Apply[Rectangle[#1+{-5/4δ,-δ/2},#2+{-δ/4,-δ}]&,stemBBs,{1}],img];

upperNoteMask[img_Image,stemBBs_List] := imageMask[Apply[Rectangle[#1+{δ/4,δ},#2+{5/4δ,δ/2}]&,stemBBs,{1}],img];

HighlightImage[
staffs[[1]],
lowerNoteMask[staffs[[1]],findNoteStemBoundingBoxes[staffs[[1]],symbolRows[[1]]]]
]

ImageMultiply[
findHalfNoteHeads[symbolRows[[1]]],lowerNoteMask[staffs[[1]],findNoteStemBoundingBoxes[staffs[[1]],symbolRows[[1]]]]
]

findHalfNotes[img_Image,symb_Image,stems_List, signature_List]:= 
With[
{mask = findHalfNoteHeads[symb]},
Map[
makeNotes[signature][#,-1]&,
Sort@Join[
ComponentMeasurements[ImageMultiply[mask,lowerNoteMask[img,stems]],"Centroid"][[All,2]],
ComponentMeasurements[ImageMultiply[mask,upperNoteMask[img,stems]],"Centroid"][[All,2]]
]
]
]

findHalfNotes[
staffs[[1]],
symbolRows[[1]],
findNoteStemBoundingBoxes[staffs[[1]],symbolRows[[1]]],
signature
]

findQuarterNoteHeads[symb_Image]:=Dilation[Binarize[DistanceTransform[MorphologicalBinarize[ColorNegate@symb,{1/3,2/3}]],3δ/8],DiamondMatrix[1]]

shortStaff

shortSymbolRow

findQuarterNoteHeads[shortSymbolRow]

HighlightImage[shortStaff,lowerNoteMask[shortStaff,noteStemBBs]]

HighlightImage[shortStaff,upperNoteMask[shortStaff,noteStemBBs]]

ImageMultiply[findQuarterNoteHeads[shortSymbolRow],lowerNoteMask[shortStaff,noteStemBBs]]

ImageMultiply[findQuarterNoteHeads[shortSymbolRow],upperNoteMask[shortStaff,noteStemBBs]]

lowerHookMask[img_Image,stems_List] := imageMask[Apply[Rectangle[#1-{δ/3,0},#1+{δ/3+1,9/5δ}]&,stems,{1}],img]

upperHookMask[img_Image,stems_List] := imageMask[Apply[Rectangle[#2-{δ/3+1,9/5δ},#2+{δ/3,0}]&,stems,{1}],img]

HighlightImage[shortStaff,upperHookMask[shortStaff,noteStemBBs]]

HighlightImage[shortStaff,lowerHookMask[shortStaff,noteStemBBs]]

detectNoteHooks[symb_Image]:= Dilation[BottomHatTransform[symb,BoxMatrix[{Floor[δ/3]+1,0}]],{{1,1,1}}]

detectNoteHooks[shortSymbolRow]

findUpperHooks[img_Image,symb_Image,stems_List] :=
Sort@ComponentMeasurements[
MorphologicalBinarize[
ImageMultiply[
upperHookMask[img,stems],
detectNoteHooks[symb]
],
{1/5,1/2}
],
"Centroid"
][[All,2,1]]

findLowerHooks[img_Image,symb_Image,stems_List] :=
Sort@ComponentMeasurements[
MorphologicalBinarize[
ImageMultiply[
lowerHookMask[img,stems],
detectNoteHooks[symb]
],
{1/5,1/2}
],
"Centroid"
][[All,2,1]]

heads = findQuarterNoteHeads[shortSymbolRow];

notes=Sort@ComponentMeasurements[ImageMultiply[heads,lowerNoteMask[shortStaff,noteStemBBs]],"Centroid"][[All,2]]

hooks = findUpperHooks[shortStaff,shortSymbolRow,noteStemBBs]

intervalCounts[data_List, intervals:{{_,_}..}]:= Map[Count[#,True]&,Outer[IntervalMemberQ[Interval[#1],#2]&,intervals,data,1]]

hookCount=intervalCounts[hooks,Map[{#,#+2δ}&,notes[[All,1]]]]

lowerNotes=Thread[makeNotes[signature][notes,hookCount]]

findShorterNotes[img_Image,symrow_Image,stemBBs_List,signature_List]:= 
Module[
{heads = findQuarterNoteHeads[symrow],notes,lowerNotes,upperNotes,hooks,hookCount},
notes=Sort@ComponentMeasurements[ImageMultiply[heads,lowerNoteMask[img,stemBBs]],"Centroid"][[All,2]];
hooks = findUpperHooks[img,symrow,stemBBs];
hookCount=intervalCounts[hooks,Map[{#,#+2δ}&,notes[[All,1]]]];
lowerNotes=Thread[makeNotes[signature][notes,hookCount]];
notes=Sort@ComponentMeasurements[ImageMultiply[heads,upperNoteMask[img,stemBBs]],"Centroid"][[All,2]];
hooks = findLowerHooks[img,symrow,stemBBs];
hookCount=intervalCounts[hooks,Map[{#-2δ,#}&,notes[[All,1]]]];
upperNotes=Thread[makeNotes[signature][notes,hookCount]];
Sort@Join[lowerNotes,upperNotes]
]

shortStaff

findShorterNotes[shortStaff,shortSymbolRow,noteStemBBs,signature]

Źródło

Serious and Not-So-Serious Image Processing Applications
Markus van Almsick
http://www.wolfram.com/broadcast/video.php?c=97&v=1044